feat(analytics): add autocapture frustration signals (click, rage click, dead click)#982
feat(analytics): add autocapture frustration signals (click, rage click, dead click)#982rahul-mixpanel wants to merge 66 commits into
Conversation
…ead click Add autocapture functionality for automatic event tracking: - Click tracking ($mp_click): Captures all touch events with semantic info - Rage click detection ($mp_rage_click): 4+ clicks within 1000ms in 44dp radius - Dead click detection ($mp_dead_click): No UI change within 500ms after click Key components: - AutocaptureManager: Main coordinator with activity lifecycle integration - TouchInterceptor: Window.Callback wrapper for touch interception - WindowSpy: Tracks all windows (dialogs, popups) via reflection - SemanticExtractor: Extracts element info from View hierarchy - RageClickTracker: Click pattern detection with spatial/temporal thresholds - DeadClickDetector: UI change monitoring with ViewTreeObserver Configuration via MixpanelOptions.Builder.autocaptureOptions() Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add test screens for validating autocapture functionality: - ComposeAutocaptureTestScreen: Jetpack Compose test screen - XmlAutocaptureTestActivity: XML Views test screen - DemoApplication: Enables autocapture during SDK initialization Test coverage includes: - $el_id resolution (contentDescription, resource ID, hash fallback) - Dead click detection (elements with no UI response) - Rage click detection (4+ rapid taps) - Excluded controls (Switch, EditText, SeekBar) - Privacy filtering (mp-sensitive, mp-no-track) Note: XML views autocapture working; Compose needs improvement. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…erent feedback Privacy filtering: - Return null (block all events) for views marked mp-sensitive or mp-no-track - Add same check for Compose via AccessibilityNodeInfo - Add debug logging when skipping sensitive elements Dead click exclusions: - Exclude EditText (keyboard appears, cursor shows) - Exclude CompoundButton (Switch, CheckBox, RadioButton - toggle animation) - Exclude SeekBar (thumb moves with drag) These controls have inherent visual feedback and should not trigger $mp_dead_click. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…ose APIs - Add ComposeSemanticHelper to extract semantics from Jetpack Compose views using Compose's SemanticsNode API directly (no reflection) - Add compileOnly dependency on Compose UI for graceful degradation - Update SemanticExtractor to detect Compose roots and use ComposeSemanticHelper - Implement $el_id resolution: contentDescription > testTag > hash fallback - Implement sensitive element detection (mp-sensitive/mp-no-track) with ancestor checking - returns SENSITIVE_BLOCKED to prevent accessibility fallback - Disable dead click detection for Compose (ViewTreeObserver cannot detect Compose UI changes) - click and rage click work correctly Note: Dead click for Compose requires semantic tree comparison (future enhancement) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Reorder property checks so EditableText is checked before OnClick. TextFields in Compose are clickable (to gain focus) but should be identified as TextField/textbox, not Button/button. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
ViewTreeObserver cannot detect Compose UI changes since Compose renders inside a single AndroidComposeView. This implements semantic tree comparison for Compose clicks: captures semantic hash at click position (text, states, bounds, children), then compares after timeout to detect UI changes. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…snapshot comparison Use semantic tree snapshot comparison for Compose dead click detection: - Capture snapshot immediately at click (nodeCount + contentHash) - At timeout, detect changes via content hash or significant node count change - Navigation detected by node count increase (new screen composables added) - Ripples don't affect semantic tree, so dead clicks are correctly detected Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add complete integration guide for autocapture feature covering click, rage click, and dead click tracking. - Add analytics/docs/autocapture.md (620 lines) - Overview and quick start with Kotlin/Java examples - Configuration options for all event types - Complete event properties reference - Element identification best practices - Full Jetpack Compose support documentation - Multi-window tracking explanation (WindowSpy) - Privacy considerations and sensitive data handling - Troubleshooting guide with debug logging - ProGuard configuration notes - Migration guide from manual tracking - FAQ section - Update analytics/README.md - Add Features section with Autocapture overview - Add quick setup examples (Kotlin + Java) - Link to detailed autocapture guide - Update table of contents Documentation provides comprehensive coverage exceeding iOS guide, with dual-language examples and complete API reference.
Add 9 instrumented tests covering: - Click event capture and property verification - Element ID resolution (contentDescription, resource ID, hash fallback) - Rage click detection with rapid touch injection - Dead click detection for unresponsive elements - Privacy filtering for mp-sensitive elements - Multiple click event generation - Token and distinct_id verification
- Add testElementIdResolutionRule3HashFallback: verifies hash fallback format (Button_view_<hex>) when both contentDescription and resource ID are absent - Add testClickEventCapturesElText: verifies $el_text captures the button's visible text content via SemanticExtractor.extractText
Add 9 instrumented tests for Jetpack Compose autocapture, mirroring
the XML test suite:
- Click event capture with contentDescription element ID
- Element ID from testTag (Compose Rule 2 fallback)
- Element ID hash fallback (no contentDescription, no testTag)
- Rage click detection with rapid sendPointerSync injection
- Dead click detection (onClick={} with no UI change)
- Privacy filtering for mp-sensitive composables
- Multiple click event generation
- $el_text capture from Text composable
- Token and distinct_id verification
Uses sendPointerSync for touch injection because compose-ui-test's
performTouchInput bypasses Window.Callback where TouchInterceptor
lives. Compose test rule is used only to find element bounds.
Remove testClickEventPropertiesComplete and testElementIdResolutionRule1 as they duplicate coverage already provided by testXmlClickEventBasic.
- Add MAX_RECURSION_DEPTH (20) to prevent StackOverflowError in countViews, computeContentHash, computeTreeHash, findNodeAtPositionRecursive, and collectTextFromChildren - Fix WeakHashMap iteration in stop() to avoid ConcurrentModificationException - Fix WindowSpy.getRootViews() returning stale data by pointing to the live delegating list instead of the replaced original - Fix ClickEvent.isComposeClick() unreliability by using a boolean flag set at construction time instead of checking WeakReference at read time - Fix AccessibilityNodeInfo leak on exception in findNodeAtPosition
tylerjroach
left a comment
There was a problem hiding this comment.
I haven't finished reviewing the code, but had these questions on autocapture doc
…tent $el_text (visible text content of tapped elements) is no longer collected by default. Users must explicitly enable it by setting captureTextContent to true in AutocaptureOptions, protecting user privacy by default.
Simplify element exclusion to a single marker. Both mp-sensitive and mp-no-track had identical behavior (full block), so only mp-no-track is needed.
…xclusion Remove text content capture entirely from autocapture. Tracking visible text can be invasive and raise privacy concerns. The complexity of nested view hierarchies can also cause text extraction to capture content from unintended views — for example, tapping a container layout might extract text from a deeply nested label unrelated to the tap. Without text capture, the mp-no-track element exclusion marker is no longer needed since the remaining properties ($el_id, $el_tag_name, $attr-aria-label, $attr-role, $elements) are purely structural UI metadata with no PII risk. Removed: captureTextContent config, $el_text property, text extraction and sanitization (CC/SSN regex), sensitive input detection, mp-no-track marker checks, and all related tests and demo elements.
- Remove redundant API 19 check in WindowSpy (minSdk is 21) - Guard autocaptureOptions(null) with warning instead of NPE - Add depth limit to findViewAtPosition for parity with Compose path - Expand dead click excluded controls docs with full platform breakdown - Add Javadoc to AutocaptureOptions.Builder clarifying defaults
|
Not to overwhelm here but some notable findings to look into from Fable. 1. Autocapture events silently dropped when
|
Compose screen embeds an XML TextView updated by a Compose button; XML screen embeds a ComposeView button that updates an XML TextView. Both reproduce the false dead click from cross-framework UI changes.
ComposeUiChangeMonitor now also monitors the XML view hierarchy so that a Compose button click which only modifies XML views (e.g. TextView text) is correctly detected as a UI change instead of a false dead click. Changes: - ComposeUiChangeMonitor takes rootView, captures XML content hash baseline, checks both Compose semantic tree and XML hash at timeout - Attach ViewTreeObserver listeners for early cancellation on XML layout/scroll - Extract computeContentHash as shared method on DeadClickDetector - Add instrumented test (AutocaptureMixedInstrumentedTest) and test activity - Remove cross-framework section from demo XmlAutocaptureTestActivity Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Reset downTime on ACTION_POINTER_DOWN and ACTION_CANCEL so the existing downTime > 0 guard rejects the subsequent ACTION_UP. Without this, two-finger gestures (pinch, rotate) produce a spurious $mp_click when the first finger lifts. Applied in both CurtainsHelper and AutocaptureManager fallback. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Passing null to clickOptions(), rageClickOptions(), or deadClickOptions() would set the backing field to null and NPE on the first isEnabled() call. Added defensive null checks matching the existing MixpanelOptions.Builder pattern. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ews" This reverts commit 22993bd.
This reverts commit e046db4.
Convert screen coordinates to window-relative coordinates before matching against Compose's getBoundsInWindow(). In split-screen or multi-window the window origin is offset from screen (0,0), causing node lookups to miss or pick the wrong element. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
attachRootViewTouchListener was a best-effort fallback for views without a PhoneWindow (Toasts, PopupWindows). In practice it captured zero real interactions: Toasts aren't clickable, and PopupWindow child views consume touches before reaching the root listener. Removing it eliminates an untracked listener leak and OnTouchListener clobbering. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Combines countViews() and computeContentHash() into a single snapshotViewTree() method that captures both view count and content hash in one pass, halving the number of tree traversals per dead click check. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
nodeCount was computed but never used by hasChanged(), which only compares contentHash. Removing it simplifies computeTreeHash to return a plain int instead of int[], eliminating per-node array allocations. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Tap duration filter: 800ms → 500ms to match iOS and platform long-press conventions - Dead click snapshot: add class name, CompoundButton checked state, and enabled state to hash; add alpha > 0 visibility check Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Route all autocapture events through the Autocapture class instead of direct track() calls. This ensures $mp_autocapture=true is set on all click events (was missing before) and exposes public methods for host apps to emit click events using ClickEvent + Builder. Key changes: - Add trackClick/trackRageClick/trackDeadClick accepting ClickEvent - Make ClickEvent, Builder, and toProperties() public for host apps - Require x, y, elementId in Builder constructor (mandatory fields) - Add null checks on all public method parameters - Replace EventEmitter interface with Autocapture reference - Extract trackAutocaptureEvent/mergeProperties shared helpers - Update AutocaptureManager and SemanticExtractor call sites Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace squared distance comparison with Math.sqrt() to match the Euclidean distance calculation used by iOS and JS SDKs. Functionally equivalent but improves cross-platform code consistency. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add cross-framework button/counter elements to both XML and Compose test screens for manual testing of dead click detection in mixed UI scenarios: - XML button → XML text, XML button → Compose text - Compose button → XML text, Compose button → Compose text Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
8 tests covering all cross-framework button→text combinations on both XML-based and Compose-based screens: - XML button → XML text, XML button → Compose text - Compose button → XML text, Compose button → Compose text Tests verify that clicks producing UI changes do NOT trigger false $mp_dead_click events. Test activities updated with mixed-framework elements (ComposeView in XML, AndroidView in Compose). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace separate XmlUiChangeMonitor and ComposeUiChangeMonitor with a single UnifiedUiChangeMonitor that auto-detects the UI framework composition and adapts its detection strategy: - Pure XML: full structural hash + ViewTreeObserver (unchanged behavior) - Pure Compose: semantic snapshot only (unchanged behavior) - Mixed framework: content-only XOR hash for XML + semantic snapshot for Compose, covering the complete UI surface without false positives from Compose ripple animations Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Pack view count + hash into a single long to eliminate int[] allocation per node during tree walks (~200 allocations per click avoided) - Replace recursive captureBaseline() fallback with inline XML capture to prevent infinite recursion on NoClassDefFoundError - Use getName() instead of getSimpleName() for class name hashing (getName() is JVM-cached, getSimpleName() recomputes each call) - Cache Compose classpath availability check to skip findComposeRoot tree walk entirely for pure XML apps Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Previously, autocapture was initialized unconditionally during SDK init and continued capturing events even after the user opted out of tracking at runtime. This violated the opt-out contract. Changes: - Guard autocapture init behind hasOptedOutTracking() check so it is not started when the user has previously opted out - Store AutocaptureOptions for later use by optInTracking() - Stop mAutocaptureManager in optOutTracking() to halt event capture - Start/recreate mAutocaptureManager in optInTracking() to resume - Add AutocaptureManager.isStarted() public getter for test access - Add MixpanelAPI.getAutocaptureManager() package-private for tests Tests: - testAutocaptureNotStartedWhenOptedOutByDefault - testAutocaptureStopsOnOptOut - testAutocaptureRestartsOnOptIn - testAutocaptureStartsOnOptInAfterOptOutByDefault Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Use platform-agnostic naming instead of web-specific ariaLabel. The API property name ($attr-aria-label) is unchanged. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
| private static ClickEvent.Builder extractFromNode(@NonNull SemanticsNode node, float x, float y) { | ||
| SemanticsConfiguration config = node.getConfig(); | ||
|
|
||
| String contentDesc = getStringProperty(config, SemanticsProperties.INSTANCE.getContentDescription()); | ||
| String testTag = getStringProperty(config, SemanticsProperties.INSTANCE.getTestTag()); | ||
|
|
||
| // Element ID resolution (matching Android plan): | ||
| // 1. contentDescription (from semantics { contentDescription = ... }) | ||
| // 2. testTag (from Modifier.testTag(...)) - equivalent to resource ID | ||
| // 3. ClassName_view_<hashCode> | ||
| String elementId = null; | ||
| String accessibleLabel = null; | ||
|
|
||
| if (contentDesc != null && !contentDesc.isEmpty()) { | ||
| elementId = contentDesc; | ||
| accessibleLabel = contentDesc; | ||
| } | ||
|
|
||
| if (elementId == null && testTag != null && !testTag.isEmpty()) { | ||
| elementId = testTag; | ||
| } | ||
|
|
||
| if (elementId == null) { | ||
| String tagName = getTagName(config); | ||
| elementId = tagName + "_view_" + Integer.toHexString(node.hashCode()); | ||
| } | ||
|
|
||
| ClickEvent.Builder builder = new ClickEvent.Builder(x, y, elementId); | ||
|
|
||
| if (accessibleLabel != null) { | ||
| builder.accessibleLabel(accessibleLabel); | ||
| } | ||
|
|
||
| // Tag name from role | ||
| String tagName = getTagName(config); | ||
| builder.tagName(tagName); | ||
|
|
||
| // Role | ||
| String role = getRoleString(config); | ||
| builder.role(role); | ||
|
|
||
| // Interactive check | ||
| builder.isInteractive(isInteractiveElement(config)); | ||
|
|
||
| MPLog.d(TAG, "Extracted Compose semantics - id: " + elementId + | ||
| ", tag: " + tagName + ", role: " + role); | ||
|
|
||
| return builder; | ||
| } |
There was a problem hiding this comment.
$elements hierarchy property missing from Compose events
extractFromNode never calls builder.elements(...), so every Compose-originated $mp_click, $mp_rage_click, and $mp_dead_click event will omit $elements entirely. Both the XML path (extractFromView → buildHierarchyString) and the accessibility fallback path (SemanticExtractor.extractFromNode → buildHierarchyFromNode) populate this field; the direct Compose SemanticsNode path does not. Any Mixpanel query or funnel that filters on $elements will silently drop all Compose-generated events.
Add clear guidance on what each field represents, how the SDK collects it, and what values to pass when tracking manually. Align documentation structure across iOS and Android. Remove button/clickable text content as elementId fallback in Compose accessibility node extraction — text on interactive elements should not be collected as identifiers.
Align Android role values with iOS: PascalCase (Button, Switch, Slider, etc.) and null when no role is detected instead of "none". Also align role names: img → Image, textbox → TextField, combobox → ComboBox, checkbox → Checkbox, radio → Radio.
Summary
$mp_click,$mp_rage_click, and$mp_dead_clickevent detection for both XML Views and Jetpack Compose UIsAutocaptureOptionsconfiguration (disabled by default, opt-in viaMixpanelOptionsbuilder) with per-feature options (ClickOptions,RageClickOptions,DeadClickOptions)Window.Callbackwrapping, withWindowSpyfor dialog/popup/menu window trackingSemanticsNodetree traversal (graceful degradation when Compose is not present)Privacy
Autocapture does not capture visible text content (
$el_text) from tapped elements. Tracking text can be invasive and raise privacy concerns. The complexity of nested view hierarchies can also cause text extraction to capture content from unintended views — for example, tapping aLinearLayoutcontainer might extract text from a deeply nestedTextViewthat isn't semantically related to the tap.The captured properties (
$el_id,$el_tag_name,$attr-aria-label,$attr-role,$elements,$x,$y) are purely structural UI metadata with no PII risk.Captured Properties
$x,$y$el_id$el_tag_nameButton,TextView)$attr-aria-label$attr-roleButton,Switch,Slider)$elementsArchitecture
TouchInterceptorWindow.CallbackSemanticExtractorComposeSemanticHelperRootForTest/SemanticsNodeRageClickTrackerDeadClickDetectorAutocaptureManagerWindowSpyTests
AutocaptureInstrumentedTest)ComposeAutocaptureInstrumentedTest)Demo app
mixpaneldemofor manual validationHardening (code review fixes)
StackOverflowErrorWeakHashMapiteration instop()to avoidConcurrentModificationExceptionWindowSpy.getRootViews()returning stale dataClickEvent.isComposeClick()unreliability fromWeakReferenceGCAccessibilityNodeInfoleak on exceptionTest plan
./gradlew :analytics:connectedAndroidTest -Pandroid.testInstrumentationRunnerArguments.class=com.mixpanel.android.mpmetrics.AutocaptureInstrumentedTest./gradlew :analytics:connectedAndroidTest -Pandroid.testInstrumentationRunnerArguments.class=com.mixpanel.android.mpmetrics.ComposeAutocaptureInstrumentedTest./gradlew :analytics:connectedAndroidTest